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Community detection helps us simplify the complex configuration of networks, but communities are reliable 
only if they are statistically significant. To detect statistically significant communities, one approach is to repeat- 
edly perturb the original network and analyze the communities. But the perturbation approach is reliable only 
if we understand how the results depend on the underlying assumptions of the perturbation method. Here we 
explore how maintaining link correlations in resampling schemes affects the significance of communities in ci- 
tation networks. We compare maintained link correlations in non-parametric article resampling with parametric 
resampling of citations that reduce link correlations in multinomial and Poisson resampling. While multinomial 
resampling maintains the variance of individual link weights and eliminates correlations between connected 
links, Poisson resampling eliminates any link correlations. For significance analysis of ranking and clustering, 
we find that it is more important to capture the variance of individual link weights than the correlations between 
link weights. We also find that Poisson resampling underestimates the variance of link weights. Therefore, 
when only link weights are available and neither article resampling nor multinomial resampling is possible, we 
suggest a simple parametric resampling scheme that generates link variances close to link variances of non- 
parametric article resampling. Nevertheless, when we highlight and summarize important structural changes in 
science, we find that the more link correlations we maintain in the resampling scheme, the earlier we can predict 
structural change. 



1. INTRODUCTION 



Researchers use network theoryl 1 1 to better under- 
stand complex systems||2H3 that have many interacting 
components! 6-1 0|. In network theory, there is a great inter- 
est in detecting communities lfTTHSTI : that is, tightly intercon- 
nected structural patterns of the network. Community detec- 
tion helps us simplify the structure of the network because the 
communities often correspond to functional units of the sys- 
tem, but communities are reliable only if they are statistically 
significant [22-25 1. Detecting statistically significant commu- 
nities is possible when we have many instances of the network 
because then we can iteratively cluster the instances to find a 
set of community descriptions. From these descriptions, we 
can assess the significance of each community. But most of- 
ten, we only have a single observation of the real network. To 
overcome this challenge and detect significant communities of 
real networks, we need a stochastically sound procedure that 
generates resamples of the single raw network. A common ap- 
proach for generating resamples of the raw network is to use 
perturbation techniques [26-29], in which researchers assume 
a model that introduces random noise to the data. The idea 
behind the perturbation approach is fairly simple: a network 
is the result of accumulating some natural events with spe- 
cific properties. By using a perturbation method, researchers 
make an assumption about the events that generate the ob- 
served network. Based on the underlying assumption, they 
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can imitate the process of network generation and create var- 
ious samples of the raw network. Afterwards, they can ana- 
lyze recreated networks and aggregate the result of the cor- 
responding communities such that they can determine which 
communities of the raw network are significant and to which 
degree. But when using the perturbation approach, we must 
ask: How much do the results of significance analysis depend 
on the perturbation method and its underlying assumption? 
More specifically, how important are the link correlations in 
the resampling scheme? 

Testing and analyzing the perturbation method is possible 
only if we have more information available about the raw net- 
work. Here we aim to explore how much the link correlations 
of the resampling schemes affect the results of significance 
analysis in the case of weighted, directed citation networks. In 
weighted networks, the significance of communities not only 
depend on the strength of the links, but also on the correla- 
tions between links: which links are simultaneously strong. 
In previous work, and with data limited to citation counts be- 
tween journals, we used Poisson resampling without link cor- 
relations to generate bootstrap networks[28 1. That is, inde- 
pendently from other links, we resampled the weight of each 
weighted directed link from a Poisson distribution with mean 
equal to the original link weight. With access to article level 
data, we now can accurately resample articles to better assess 
the significance of communities under the condition that link 
correlations are fully preserved. In addition, we can better un- 
derstand the effects of eliminated link correlations in Poisson 
resampling. Our dataset includes citations between more than 
900,000 scientific articles published in about 1 1,000 journals 
in the years 1984-2010. For a specific year, we can build a ci- 
tation network between scientific journals in which each link 
weight between two journals A and B represents the number 
of times that articles published in journal A cite articles pub- 
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lished in journal B. 

With the article level citation data, we can explore how link 
correlations in different resampling schemes affect the statis- 
tical properties of communities. But because communities are 
aggregated measures of a network, in order to better under- 
stand the role of the resampling scheme on the statistical anal- 
ysis of communities, we will investigate how much resam- 
pling schemes affect the finest detail levels: individual links. 
We also will investigate what role the link correlations of the 
resampling scheme play on the properties of neighbor links: 
links that share a common source journal. The link variance 
and link correlations contribute to the statistical analysis of 
aggregated measures of a network such as clustering or rank- 
ing. 



2. METHODS 

To understand which effect the link correlations of the re- 
sampling scheme have on assessing significant communities, 
we compare a resampling scheme with full preserved link cor- 
relations (article resampling), a resampling scheme with half 
preserved link correlations (multinomial resampling) and a re- 
sampling scheme with no preserved link correlation (Poisson 
resampling), see Fig.[T] Keeping half of the link correlations 
in multinomial resampling corresponds to conserving within- 
link correlations and destroying between-link correlations. 

In the first scenario, if the article level citation is available, 
the scientific process that eventually results in the citation net- 
work is either an article published or not. We can actually re- 
sample from real articles and assemble the bootstrap networks 
by aggregating articles in journals. The process of article re- 
sampling is simple: assume that we have a pool of all articles 
that participate in our citation network. We pick an article 
from this pool and add the citations of this article from the 
journal in which the article was published to the cited jour- 
nals. Then we put this article back in the pool. We continue 
this process until we reach the same number of articles that 
the original network has. Since one article might cite arti- 
cles from different journals, the event of article resampling, 
automatically introduces correlation between the links of the 
bootstrap networks (Fig.[T|a)). For example, suppose that an 
article x that was published in PNAS cited an article in Na- 
ture and also an article in Science. When article x is chosen 
in the resampling process, it increases the link weight from 
PNAS to Nature and the link weight from PNAS to Science 
concurrently. In addition, because an article might cite arti- 
cles of a specific journal more than once, article resampling 
introduces correlations within links (Fig. [TJa)). To investi- 
gate how these correlations affect the significance analysis, 
we compare article resampling with multinomial resampling, 
which keeps correlations within links but destroys correlations 
between links. 

In the second scenario, we consider a situation in which 
the full article level citation is not available but the detailed 
information about the events that generate a specific link is 
known. This case naturally suggests resampling every link 
weight based on a multinomial resampling. To generate boot- 
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FIG. 1: Link correlation preservation in different resampling 
schemes (a) Article resampling preserves correlations between links 
and also correlations within links. For example, an article in journal 
Jl might cite articles from journal J2 together with articles in journal 
J3 (correlations between links). An article in journal Jl might also 
cite another journal J2 more than once (correlations within links). 
The right hand side shows some examples of possible resampled net- 
works that necessarily keep correlation between and within links, (b) 
Multinomial resampling only preserves the correlations within links. 
The examples of resampled networks on the right hand side shows 
that they could be generated without keeping between-links correla- 
tions, (c) Poisson resampling does not preserve any link correlation. 
Every link of a resampled networks is generated independently of 
others. 



strap networks, we resample every link weight of the raw 
network from a multinomial distribution, which is estimated 
based on article level data information, and we do this inde- 
pendently for all links of the raw network. Multinomial re- 
sampling does not preserve correlations between links, but it 
does preserve the correlations within links (Fig.[TJb)). As are- 
sult, multinomial resampling creates an intermediate stage be- 
tween a completely destroyed link correlation scheme (Pois- 
son resampling) and a fully preserved link correlation scheme 
(article resampling). The question is: how much is the result 
of significance analysis in the reduced-link correlation scheme 
(multinomial resampling) different from the fully preserved- 
link correlation scheme (article resampling). In section[3] we 
will show that significant clusters in multinomial resampling 
are close to significant clusters in article resampling, which 
implies that the role of link dependency on significance anal- 
ysis of clusters is small. 

In the third scenario, if we only have access to the jour- 
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nal citation data, we can assume that the scientific process 
that eventually results in the citation network is either a re- 
searcher adds a citation between two journals or not. This 
event is equivalent to assuming that citations are spread ac- 
cording to a Poisson distribution. In such a case, the event 
of citation between a pair of journals is independent of cita- 
tion between another pair. The process of Poisson resampling 
for generating bootstrap networks is as follows: we resample 
every link weight of the raw network from a Poisson distribu- 
tion with mean equal to the original link weight, and we do 
this independently for all links of the raw network. Poisson 
resampling not only automatically ignores the correlation be- 
tween links but it also ignores the correlations within links: 
Poisson resampling assumes independent events that partic- 
ipate in making a specific link (Fig. |TJc)). The question is: 
how much do the results of significance analysis specifically 
depend on the independent link weights assumption? We aim 
to investigate which role these correlations play in the boot- 
strap networks and what effect these differences would have 
on the significance analysis of the clusters. 

3. RESULTS AND DISCUSSION 

In order to investigate the effect of link correlations on the 
significance analysis of clusters, we create 1000 bootstrap net- 
works based on a resampling scheme. Then we search for sig- 
nificant clusters, cluster cores, which we define as the biggest 
subset of nodes in each cluster that gathered together in more 
than 90% of the bootstrap networks. For clustering, we use 
infomap, which is an information theoretic algorithm that re- 
veals regularities in a given network based on how informa- 
tion flows on that network [30]. Figure[2]shows the difference 
between significant cluster cores of article, multinomial, and 
Poisson resampling in terms of normalized information dis- 
tance. Normalized information distance is defined as: 

I(C,C) 

a,nax ~ max(H(C),H(C')) () 

where H(...) refers to Shannon entropy and I(C, C) is the mu- 
tual information between the significant cores of the two re- 
sampling schemes that tells us how similar they are. Mutual 
information between two clusters C and C is described as: 

ZP(c,c') 
P(c,c') log \ p (2) 
P(c)P(c') 

c,c' 

where P(c, c') is the joint probability distribution between two 
clusterings c and d . P(c) and P(c') refer to the marginal prob- 
ability distributions. The second term in Eq. |T| is called nor- 
malized mutual information. 

If C and C are identical, then the normalized mutual infor- 
mation is equal to 1, which means that, by knowing one cluster 
structure, we know the other one, and conversely, if C and C 
are completely independent, by knowing one, we learn noth- 
ing about the other one and the normalized mutual informa- 
tion between them would be 0. We use normalized informa- 
tion distance for comparing clusterings because it is a sound 
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FIG. 2: The differences between significant clusters' cores in 
different resampling schems. We calculate normalized information 
distance (d maw ) between the significant cores of the two correspond- 
ing methods with respect to the page rank. All values correspond to 
an average over at least 2000 runs. 

metric[ 3 1 1 ■ Figure[2]shows that the difference between signifi- 
cant cores of article and multinomial resampling is in the same 
order as the difference between each of these schemes with it- 
self, and both of them are considerably different from Poisson 
resampling. Although multinomial resampling does not hold 
the correlation between citations and article resampling does, 
our results show that between-link dependency does not have 
a great impact on the significance analysis of clusters. 

We further explore the effects of link variance in a concrete 
example. Figure [3] shows the alluvial diagram of the three re- 
sampling schemes over years 1989-1993. In the alluvial dia- 
gram, each block represents a specific module in a given year. 
The height of a block represents its importance in terms of 
PageRank 021 . In a block, the lighter colors correspond to 
the insignificant part of the module; the bigger this area is, the 
more insignificant nodes that module has. The white verti- 
cal gap between blocks determines separate modules, and the 
numbers under each block correspond to the year. As shown 
in the figure, all three resampling schemes agree on the sep- 
aration of Nuclear & Particle physics from General physics 
as an independent stand-alone module in 1993. In fact, given 
the data we have, we see the separation of Nuclear & particle 
physics as a significant stand-alone module in 1993, but by no 
means we conclude that this is the emergence of a new field. 
While Nuclear & particle physics was considered a research 
area long before 1993, it takes some time before it shows up 
in the structure of the network. The separation of a module 
in citation network data reflects the specialization of scien- 
tific fields according to the journal's citation pattern. But as 
Fig.[3]shows, different schemes consider the separation of the 
Nuclear & particle physics module differently in time. For 
example, in article resampling, this module is a completely 
insignificant part of General physics even in 1989, while in 
Poisson and multinomial resampling, this happens later. So 
the process of becoming insignificant could provide us a sig- 
nal about important changes that might happen in the future; 
apparently, article resampling can give this signal sooner than 
multinomial resampling, and multinomial resampling can give 
it sooner than Poisson resampling. In summary, for signifi- 
cance analysis of communities, within-link correlations play 
a more major role than between-link correlations. Conserving 
link correlations in a resampling scheme helps us to recog- 
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FIG. 3: The separation of Nuclear and particle physics from the physics module. While all three resampling schemes agree on the 
separation of Nuclear & particle physics from General physics into an independent stand-alone module by 1993, article resampling emits a 
signal about this change sooner than multinomial or Poisson resampling. 
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FIG. 4: The variation of the PageRank for top rank journals based on different resampling schemes. In agreement with the result of 
single link variance analysis, our analysis shows that core structures in article and multinomial resampling are much more similar to each other 
than in the Poisson resampling. The article resampling is the biggest perturbation, in which the 95% confidence interval for the PageRank is 
broader than in multinomial or article resampling. Multinomial and article resampling were second and third, respectively. 



nize the changes of a network earlier. In another words, as we 
neglect link correlations in our resampling, we postpone the 
emergence of insignificant clusters further. Consequently, we 
postpone a signal that could give us a hint about changes that 
might happen in the future. 

We also investigate PageRank as another aggregated mea- 
sure to test and analyze the effect of a resampling scheme and 
its correlation strategy on the network. In calculating PageR- 
ank, the importance of a node (a journal in our citation net- 
work) corresponds to the importance of nodes that cite this 
node. So the full network indirectly participates in calculating 
the PageRank of a node. Figure|4]shows how much the PageR- 
ank of some top journals would vary based on the resampling 



scheme. The length of each line accords with an interval that 
covers the variation of PageRank for a given journal in a given 
resampling scheme. The numbers on the left/right hand side 
of each line correspond to the minimum/maximum rank or- 
der of each journal for a resampling scheme. Science has the 
largest PageRank value in the raw network, and so it is the first 
journal in the rank order. In the Poisson resampling, Science 
always maintains its first rank position in the ranking list. But 
in multinomial and article resampling, sometimes the rank of 
Science decreases to the second position. In a similar fash- 
ion, the rank order of PRL {Physical Review Letters), NEJM 
(New England Journal of Medicine) and J Neurosci ( Journal 
of Neuroscience) is changed based on the resampling scheme 
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that is used. In general, the PageRank of a node varies more 
in article resampling compared to Poisson or multinomial re- 
sampling. In this respect, we study the effect of resampling 
schemes on the rank order of all nodes in the network. We 
sample pairs of nodes (z, j) from the rank order that we ob- 
tain from a resampling scheme and compare them with the 
rank order that we obtain from another resampling scheme. 
We sample pairs of nodes proportional to their PageRank and 
measure the similarity between the two rank orders in terms 
of normalized mutual information. If for all possible pairs 
in the two-rank order, the node with the highest rank in one 
order is the same in the other order, the mutual information 
between the two rank orders would be one. The more dif- 
ferent the two rank orders are (less common pair orders), the 
smaller the MI between them would be. If the two rank orders 
do not have any common pair orders, the MI between them 
would be zero. In a quantitative analysis of the rank order for 
the different resampling schemes, we find that the normalized 
information distance (Eq. [TJ between two different rankings 
that are generated with the same resampling scheme, on av- 
erage, is about 26 percent larger for article resampling than 
for Poisson resampling and 23 percent larger for multinomial 
resampling than for Poisson resampling. For ranking, arti- 
cle resampling has the biggest variation, but multinomial re- 
sampling without correlations between links varies almost as 
much as article resampling. Multinomial resampling can ex- 
plain almost all ranking variances of article resampling with 
correlations between links. 

The link dependency strategy of a resampling scheme does 
not have a great impact on ranking (Fig. [4} or on the signifi- 
cance analysis of clusters (Fig. [2}. The reason it doesn't have 
a great impact is manifested in the behavior of resampling 
schemes at the detailed link level. 

To better understand the results of resampling schemes on 
aggregated measures (ranking and clustering), we investigate 
what role the link correlations of a resampling scheme play 
on the properties of neighbor links. We investigate whether a 
preserved link correlation scheme, e.g, article resampling, has 
an expressive effect on the correlation between link weights or 
not. Then we look at the variation of individual links on each 
resampling scheme. Our results show that the multinomial re- 
sampling completely matches with the article resampling on 
the link level. Where the probabilities of different link weights 
are unknown for the purpose of performing multinomial re- 
sampling, we propose a simplistic model that can estimate the 
probabilities. 



there is a slight correlation between neighbor links. To check 
if this is a significant correlation or not, we compare article re- 
sampling with its close acquaintance which has independent 
assumptions between links: multinomial resampling. Figure[5] 
shows that when we compare article and multinomial resam- 
pling, the correlations between neighbor links are small. In 
fact, we could say that most neighbor links are not correlated. 
Those neighbor links that are slightly correlated tend to be 
correlated rather than anti-correlated. As we saw in the begin- 
ning of section [3] this slight correlation doesn't have a great 
impact on the significant cluster cores or ranking of nodes. 
As shown, the dependency between links has a small effect on 
significant cluster cores, but nevertheless it influences the time 
that insignificant clusters emerge and can give a clue about 
important changes that might happen in the future. 
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FIG. 5: Article resampling does not strongly preserve the de- 
pendency of neighbor links. The correlation distribution for a pair 
of neighbor links where at least one of them has a specific weight. 
By definition, article resampling introduces correlation to the neigh- 
boring links and multinomial resampling ignores any correlation. By 
comparing, we see that the result of correlation distribution confirms 
that most correlations of article resampling are not significant, when 
we compare them with the multinomial resampling as null mode. All 
points correspond to an average of at least 50 runs. 



3.2. Single link variance 



3.1. Pairwise link variance 

Article resampling introduces dependencies between link 
weights: an article can cite papers in different journals, so 
picking that article adds citations to more than one journal 
simultaneously. Here we want to measure how much these 
neighbor links are correlated in the resampled networks. We 
call two links neighbor links if they share a common source 
node. Figure [5] shows that in article resampling, for most 
cases, neighbor links are not correlated, but in some cases, 



Figure [6] shows how much a specific link weight, w*, varies 
based on the resampling scheme. When the link weight is 
very small, for example, w* = 1, we see that the probability 
distribution of link weights in Poisson resampling perfectly 
matches with the article resampling (Fig. |6ja)). The link 
weight equal to one means that only one article contributes 
to the citation between two journals, so the chance of pick- 
ing that article is A, where N is the total number of articles. 
Therefore, after resampling N articles, the chance of getting 

that specific paper k times is h (^r-)^ N ~ k \ which, in the limit 
of large N, coincides with the definition of Poisson(l). But 
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FIG. 6: Comparing the probability distribution of link weights in article resampling with Poisson resampling and multinomial resam- 
pling (a) For low link weight (w* = 1), the Poisson resampling precisely coincides with article resampling. (b,c) For medium values of link 
weight (w* = 10) and high values of link weight (w* = 100), the Poisson resampling underestimates the article resampling. The variance of 
the distribution in article resampling is much higher than in Poisson resampling. For example, for w* = 10, the variance of article resampling 
is a 2 an = 19, while the variance of Poisson resampling is <ri oiss = 10. Similarly, for link weight w* = 100, the variance of article resampling 
is cr 2 rt = 290, while the variance of Poisson resampling is cr 2 pojss = 100. The variance of multinomial resampling is quite close to the article 
resampling, which confirms that the multinomial model imitates article resampling and make the distribution broader than Poisson resampling, 
(d) The variance of link weights in article resampling and Poisson resampling averaged over all resamples. All points correspond to averaging 
over 1000 runs. 



when the link weight between two journals is higher than one, 
for example, medium values like w* = 10 in Fig.[6|b) or high 
values like w* = 100 in Fig.[6jc), we see that the variance of 
Poisson resampling underestimates the variance of article re- 
sampling. This happens because citations can come in groups: 
for a link where its weight w* is medium/high, there are A ar- 
ticles ( A <= w*) that contribute to that weight, and some- 
times articles might add more than one citation. So, although 
article resampling gives the same average weight as Poisson 
resampling, the variance of that weight in article resampling 
would be higher than for Poisson resampling. In summary, 
high weights cause greater differences between the variance 
of article resampling and Poisson resampling (Fig.[6jd)). 

Indeed, although Poisson resampling assumes an enormous 
amount of binomial events that produce a specific link weight, 
article resampling tells us that the observed link weight is the 
outcome of multinomial events. In multinomial resampling, 
every link weight is generated from a multinomial distribution 
independently from other links. Although multinomial resam- 
pling assumes independency between links' weights and arti- 
cle resampling does not, Fig. |6jb,c) shows that multinomial 
resampling completely matches article resampling on the link 
level. Multinomial resampling intrinsically considers group 
citations, and therefore it can generate higher variance than 
Poisson resampling. But what if the probabilities of different 
link weights are unknown for a given network? To estimate 
the probabilities, we look at the number of papers that con- 
tribute to a link with a specific weight. Figure |7|a) shows 



that when link weight w is high, the number of papers that 
contribute to generating that link weight Np(w) is far from 
the value of the weight itself. Figure [TJb) shows that, when 
the link weight increases, the fraction of single citations that 
contribute to making that weight is reduced. To generate 
higher link weights, more group citations (2,3,... citations) 
contribute. As Fig. |7|a) shows, the number of papers that 
contribute to generating a link weight w scales as w 9 for all 
years. We use this information to build a model for estimating 
the multinomial distribution when the probabilities of differ- 
ent link weights in a given network are not known. For each 
weight w, we assume that it is generated out of papers with 
only one or two citations. We can simply estimate the number 
of papers with one citation N\ and the number of contributing 
papers with two citations Ni by solving the following linear 
equation system: 

N P {w) = Ni+N 2 = w 09 ) Ni = 2w 9 - w 

=> (3) 
N\ +2N 2 = w\ N 2 = w- w - 9 

After estimating Ni and N 2 , we suggest resampling every link 
weight by using the following minimal model: 

Poisson(Ni) + 2Poisson(N 2 ) (4) 

The variance that we could get out from this model is: 

Var(Poisson(Ni) + 2Poisson{N 2 )) = 

o 0.9 (-^ 
3w - 2w 
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FIG. 7: The high variance of article and multinomial resampling can be estimated by a simple model that extends Possion resampling 
to account for papers that contribute with multiple citations to the same journal, (a) Average number of papers that contribute to a specific 
link weight in logarithmic scale. For all years, the number of papers with weight w (N P {w)) fits to the function x" with exponent a = 0.9. (b) 
The fraction of 1, 2, 3 and 4 citations that contribute to building a specific link weight w. Compared to low link weights, high link weights 
have a lower fraction of papers with only one citation and a higher fraction of papers with 2, 3 or 4 citations, (c) The probability distribution of 
link weight w" = 10 for 4 cases: Poisson resampling, article resampling, multinomial resampling, and the minimal model. The high variance 
of article/multinomial resampling could be estimated by the model, (d) The model can generate higher variance than Poisson resampling for 
different link weights. However, it could not generate exactly as high a variance as article resampling. 



In Fig. |7|c), we show the probability distribution of link 
weight w* = 10 for four cases: Poisson resampling, article 
resampling, multinomial resampling, and the proposed model. 
As shown, the high variance of article/multinomial resampling 
could be estimated by the minimal model. However, this esti- 
mation is not exact because the minimal model does not take 
into account group citations with three citations or more. In 
summary, the model can generate higher variance than Pois- 
son resampling for different link weights, but it can not gener- 
ate exactly as high a variance as article resampling (Fig.[7|d)). 

4. CONCLUSION 

Link correlation of a resampling scheme influences the sig- 
nificance analysis of communities and ranking. We compare 
three scenarios: full preserved link correlations (article re- 
sampling), half preserved link correlations (multinomial re- 



sampling), and no preserved link correlations (Poisson resam- 
pling). We found that the result of significance analysis in 
multinomial resampling that only conserves correlation within 
individual links almost matches with article resampling that 
conserves correlations both within and between links. We 
conclude that the role of correlation within individual links is 
greater than the role of correlation between links. Neverthe- 
less, we found that conserving link correlation in a resampling 
scheme can provide an early hint of possible changes of the 
network in the future. These findings can help researchers to 
better understand and asses reliable significant communities 
and structural changes for a given network. 
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